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Remarks 

Claims 1-11 are 
is the independent claim, 
remarks is respectfully requested 

The examiner objecteji 
form because a multiple 
only. It is respectfully poimefl 
5 makes reference to sub-i 
1 and 4, respectively. HoweMer, 
depends from claim 1 . 
incorporation, and claim 5 is 
should be withdrawn. 

The examined rqeclet 
Mehrotraet al. 

Independent claim 1 
optimize the structure of the 
having a plurality of input 
layer having a plurality of in 
output neurons that provide 
synapse inteicounects two 
the input layer toward the output 
eliminating synapses that 
First and second sending neutons 



I pend^g in the application. Claims 1-11 were rejected. Claim I 
Reconsideration of the rejection in view of the following 



to claim 5 under 37 CFR L75(c) a.*; being in improper 

claim should refer to other claims in the altemative 
out that claim S is not a multiple dependent claim. Claim 
al-3) and bl'3), which are originally recited in claims 
\ claim 5 dq)ends directly from claim 4, which in turn 
1, each of these sub-elements is recited in claim 5 by 
lot a multiple dependent claim* The objection^ therefore. 
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nei irons i 



claims 1-1 1 under 35 USC §102(b) as being anticipated by 

rbcites a method for training a neural network in order to 
i^eural network. The neural network includes an input layer 

that receive the input data, at least one intemiediate 
t^rmediate neurons, an output layer having a plurality of 
output signals, and a multiplicity of synapses, wherein each 
of difTcrent layers, defining a sending direction from 
layer. The training method includes identifying and 
no significant influence on the curve of Hie risk function, 
are selected that are connected to the same receiving 



ne irons i 



have 
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changed 1 



neurot) by respective first anc 
response signals &om the first 
The first synapse is interrupt<|d 
The output signals of the 
the unchanged neural networl : 
level, the first synapse is eli 
network. 

In contrast, Mehiotra 
process. Mehrotra et at first 
network, one of which is to 
that can be considered 
al. describe four pruning 

1. Elimination 
elimination of nodes having 

2. Elimination 
error. The test for these 
zero or some other weight 

3. Elimination of inpi^ 
negligible. 

4. Determination of ^ 
derivatives of the error fimctipn 

All of Che disclosed 
with the goal of optimizing 



Pages 



second synapses. It is assumed that there is a correlation of 
and second sending neurons to the same receiving neuron, 
and a weight of the second synapse is adapted in its place. 

neural network are compared with the output signals of 
Tf the comparison result does not exceed a predetermined 
elin^inated, thereby simplifying the structure of the neural 

^t al. disclose a conventional neural network pruning 
lisclose three approaches to building an optimally-sized 
ppme the network by elimination of nodes and connections 
(page 1 16, lines 6-9), in Section 4,2. 1, Mehrotra et 
iures in more detail^ as follows. 

associated with wei^its of small magnitude, and 
sjssociated connections with small magnitude weights. 

that do not significantly affect network outputs or 
consists of replacing the connection weight with 
detecting ttie change in network output 
nodes if the resulting change in network output is 



I unimp >rtant 



iprocedi 



of com Lections i 



of com Lections 1 



conn ections < 



ml 



hich weights can be eliminated by examining second 
contained in the Hessian matrix, 
probedures eliminate nodes and/or connections from a network, 
n 5twoik size. However, in determining which nodes and 
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connections to eliminate, 
element being considered, 
function. None of these 
that are connected to the sami ^ 
assuming a correlation 
the same receiving neuron^ ii 
weight of the second syn^se 
changed in accordance with 
neural network* and if the 
eliminating the first synapse, 
of these actions are required 

In the detailed 
selecting first and second 
neuron by respective first ant 
at lines 31 and 32, Fig. 1.14 
page 1 1 6 merely states that 
weights can be eliminated, 
sending neurons connected 
process require more than 
pruning to take place. This 

The examiner 
of response signals from saic 
neuron on page 19, in Fig. 1 



Mel irotra < 



en pirical 



liei 



coi ipanson 3 



[Tejecti(»n, 
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et al. rely on an assessment of the weight of the 
results of elimination, or examination of an error 
lures includes selecting first and second sending neurons 
receiving neuron by respective first and second synapses, 
signals from said first and second sending neurons to 
ir^jlerrupting the first synapse and adapting in its place a 
comparing the output signals of the neural network 
previous action with the output signals of the unchanged 
result does not exceed a predetermined levels 
thereby simplifying the structure of the neural network. All 
>y claim 1, and Mehrotra et al. do not disclose any of them, 
the examiner asserts that Mehrotra et aK disclose 
g neurons that are connected to the same receiving 
second synapses on page 19» in Fig. 1,14, and on page 1 16, 
shows the layout of an acyclic network. The passage on 
n odes having associated connections with small magnitude 
passage does not disclose selecting first and second 
the same receiving neuron, nor does the disclosed pruning 
sending neuron connected to a receiving neuron for 
<{laim element is not disclosed by Mehrotra et al. 

that Mehrotra et al. disclose assimiing a correlation 
first and second sending neurons to the same receiving 
14, and on page 46, in Fig. 2.5. Fig. 2.5 shows a generic 



Ihei 



one 



further asserted < 
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ncdcs 



This 



:of diffeieni 



perceptrot) for nnlimensional 
of the products of all input 
produces a step function, 
can separate samples 
dimensions, for purposes 
that the disclosed relation mij 
second sending neurons, it is 
116-118. This claim element 

The examiner asscrtcc 
and adapting in its place a 
page 117, in Fig. 4»6. The 
result of the removal of the 
algorithm. According to this 
which does not penalize perfc 
connection is deleted. The 
adapting in its place a weight 
require iirst and second 
connection to make a pruning ; 
examiner's note that removal 
second synapse. This claim 

The examiner also 
signals of the neural network 
output signals of the unci 



I wei; 



ex iiminer i 



syna] »ses 



changed 



Pages 

input space, impl^enting fte hyperpiane in which the sum 
and their respective weights is equal to zcro» which 
figure in particular is used to illustrate a hypeiplane that 
classes for spaces of higher numbers of inpul 
of t aining a network, not for pruning a network. To the extent 
J jht suggest a ccHrelation of response signals &om first and 
tiot applicable to the pruning procedure described on pages 
is not disclosed by Mehrotra et al. 
that Mehrotra ct al. disclose interrupting the first synapse 
ght of the second synapse on page 1 9, in Fig. 1 . 14, and on 
noted thai the weight of the second synapse is the 
$ynap$e. Fig. 4.6 discloses a generic network pruning 
algorithm, a node or connection is found, the removal of 
X ^rmance beyond desirable tolerance levels. The node or 
algorithm does not disclose interrupting a first synapse and 
of the second synapse. Actually, the algorithm does not 
at all, and only describes analysis of a single node or 
determination. Further, there is no support for the 
9f a first synapse results in replacement by the weight of a 
Element is not disclosed by Mehrotra et al. 
asjerted thai Mehrotra et al. disclose comparing the output 
changed in accofxlance with the previous action with the 
neural network on page 19, in Fig. 1 .14, on page 116, at 



fi^t 
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lines 33 and 34, and on page 



page 118 merely states that a 
elimination results in an erro r 
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1 17, in Fig. 4.6. The passage on page 116 merely states that 
connections having an existence that does not significantly affect network output or error 
may be pruned. That is, the c itcd passage discloses a goal of pruning^ but does not 
disclose the process of elimir ating syniq)ses recited in claim 1 . As previously noted^ Fig. 
4,6 discloses an algorithm thi X includes elimination of a node or comiection such that the 
elimination does not penalize performance beyond a desirable tolerance level. Again» 
this algorithm represents a gc al of a pruning procedure, but does not disclose the process 
of eliminating synapses reciti d in claim L This claim element is not disclosed by 
Mehrotra ct al. 

The examiner also asi erted that Mehrotra et al. disclose eliminating the first 
synapse if the comparison res ult does not exceed a predetermined levels thereby 
simplifying the structure of t1 le neural network on page 1 9> in Fig. 1.14 and on page 1 1 8 
at lines 1-5. The passage on jage 1 1 8 is a small portion of a description of the Lccun, 
Denker» and SoUa Optimal B rain Damage method of pruning a network, and therefore is 
unrelated to the other passagi ^ cited by the examiner in the Mehrotra ct aL reference. 
That is, this cited action is n< t part of the same process described in the eariier passages, 
and therefore, taken out of c( ntext, should not be q>plied against a single element in 
claim 1 when other processes \ have been applied against other elements of the claim, in an 
attempt to demonstrate antic: pation of the claimed invention. In any case, the passage on 



connection maybe eliminated if the result of the 
that is no greater than that expressed by the formula on line 
3. That is, a threshold tolera ice is expressed for pruning network connections. Again, 
this is just a goal of the prun ng process, albeit this time with a specifically-stated 
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£nd ] 



tolerance level. However, 
elimination of a connection, 
of a second synapse in place 
element is not disclosed by 

In summary, Mehrotr j i 
and that it i$ preferable to est^biish 
procedure. However, Mehrollra 
and elimination method recit4d 
connections and nodes based 
these singular eliminations, 
and comparison actions recited 
receiving neurons having at 
allow pruning to take place 
necessary or performed accoijding 

In view of the discussion 
not anticipate the invention recited 
therefore also are not anticip 
therefore, should be withdravfn 

The examiner 
response to the previous 

The examiner pointec 
that it is common for at least 
neuron, and that conelation 



address es 



1 Office 
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tolerance level is related to an error resulting ftom 
not related to an error resulting from adapting the weight 
<bf that of a first synapse, as recited in claim 1. This claim 
Nfehrotra et al. 

et al. disclose that pruning of networks is advantageous, 
ish a tolerance level for errors resulting from the pruning 
et al. do not disclose the particular synapse identification 
in claim L Ratlier, Melirotra et al. disclose elimination of 
on their weights alone, and anal)^ of the errors caused by 
1 4ehrotra et al. do not perfonn the selection, intcmiption, 
in claim 1. The process recited in claim 1 is limited to 
l^t two connected sending neurons, whareas Mehrotra et al. 
any node, because comparison between two synapses is not 
to Mehrotra et al. 
set forth above, it is submitted that Mehrotra et al. do 
in claim 1 . Claims 2-1 1 depend from claim 1, and 
ted by Mehrotra et at. The rejection of claims 1-11, 



comments particularly to the remarks submitted in 
action, which are discussed below, 
out that neural networks are massively parallel devices, and 
two sending neurons to be connected to a third, receiving 
s mong sending neuroris is shown by Mehrotra et al. This is 
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not disputed. However, thee 



Pages 



aimed invention requires selection of two particular 
sending neurons having corre ated response signals as part of the synapse identification 
and elimination process. Mebrotra et al. acknowledge that such neurons exists but do not 
perform the claimed selectior action when pruning the network. That is, altiiough 
Mehrotra et al. acknowledge >luralities of correlated smding neurons, that characteristic 
is irrelevant to the disclosed i Tuning process, whereas it is essCTitial to the claimed 
process. 

On page 8 of the actio n, the examiner notes that, according to Mehrotra et al., if 
by chance a connection is elii ninated and the receiving neuron only receives two 
connections, then necessarily the only remaining element to the receiving neuron is the 
other perceptron, which is ini luenced by the weight of the second perceptron. Accepting 
this proposition for the sake c f discussion, the elimination of one of only a pair of 
sending connections would oi \ly happen by chance according to the Mehrotra et aL 
process, whereas the claimed process explicitly includes a selection of sending neuron 
pairs. No such selection takes place according to Mehrotra etal. Rather, connections and 
nodes are selected according :o their weights, so that connections associated with weights 
of small magnitude are elimii laicd. Selection of sending neuron pairs is not disclosed by 
Mehrotra et al. 

the examiner concluded that Mehrotra ct al., at paragraph 
4.2. 1 and Fig. 4.6> disclose el imination of nodes having a weight below a threshold. This 
is the same conclusion in the applicants' argument, which the examiner quoted above his 
response. That is, Mehrotra < t al. select connections for elimination based on connection 
weight, not on analysis based on selected sending neuron pairs. The claimed invention 



At the bottom of page 
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1 examii icr 



[therefore 



does not select connections b^ised 
irrelevant to Mehrotra et al., 

On page 9, the 
plurality of neurons and 
will be selected for weight 
Mehrotra et al. In fact, Mehrotra 
connection at a time. Mehrotba 
on weight, and do not disclos ; 
It is clear from the Mehrotra 
This remaik also applies to 

It is acknowledged 
their broadest reasonable 
interpretation of **selecting 
same receiving neuron by 
cover a process in which onl) 
though each might coincidenjally 
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on weight; likewise, the existence of neuron pairs is 
Although it is required by claim t . 

asserts that the Mehrotra et ah network is made of a 
there is a plurality of pairs of neurons and such pairs 
adaptation. However^ this selection is not disclosed by 
et al. never disclose selection of more than one 
et al. disclose selection of individual connections based 
selection of pairs for any reason under any circumstances, 
al. disclosure that pairs exists but they are not selected, 
examiner's comments on page 1 0* 
the examiner has full latitude to interpret the claims in 
« However* it is respectfully submitted that no reasonable 
and second sending neurons that are coimected to die 
ive first and second synapses" could be so broad as to 
individual neurons arc selected according to weight, even 
be one of a first and second neuron. 
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respective 
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Based on the foregoing, 
overcoTne, It is therefore requested 
issue. 



March 10. 20QS 
Date 



TMC:hlp 



Page 10 

it is submitted that all objections and rejections have been 
that the claims be allowed^ and the case passed to 

Reqiectfully submitted> 



Thomas M, Champagne 
Registration No. 36,478 
IP Strategies 
12 /a Wall Street 
Suite I 

Asfaeville, North Carolina 28805 

828.253.8600 
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